The Wyner-Ziv approach to distributed source coding

In a communication network, suppose there are a collection of (correlated) source nodes that want to communicate to a common sink node, then the distributed source coding problem is one of designing a system (of encoders and decoder) that optimally compresses the correlated sources. The key aspect of the design is that the source nodes aren’t allowed to communicate with one another –i.e., the sources are encoded separately, but are decoded jointly. (If they could communicate, then the problem reduces to the single user source coding problem.) The goal of the distributed source coding problem is to design a system that approaches the optimal performance that can be obtained if the source nodes could communicate with one another. Slepian and Wolf (SW) showed that in the case of two discrete sources X and Y , it is theoretically possible to achieve the same level of performance in a system where the encoders for X and Y are separated as in a system where they are not. More formally, suppose X and Y are two statistically dependent discrete random processes, taking values in finite alphabets1, which are encoded by two separate encoders, but are decoded by a joint decoder. Then the SW result says that even if the encoders are independent, to achieve reliable transmission of both X and Y , the rates of transmission must be such that RX ≥ H(X|Y ), RY ≥ H(Y |X), and RX + RY ≥ H(X,Y ). Wyner considered the following specialization to the above problem: suppose the decoder is primarily interested in reconstructing X, and Y is considered as some side-information on X available to the decoder. Then the problem specializes to the optimal rates of sending X and Y so that the decoder can reconstruct X with high reliability. The design of such a system will be called the zero-error source coding problem with side information [1], since the system is to be designed such that there is an arbitrarily small probability of error in the reconstruction of X. (If Y is transmitted at a rate RY ≥ H(Y ), then from the SW result, it must be theoretically possible to transmit X at a rate RX = H(X|Y ) + , for arbitrarily small.) The problem is stated more formally in Section 2. A counterpart to the above problem is the lossy source coding problem, introduced by Wyner and Ziv [2]. Suppose Y is readily available to the decoder, then a system must be designed that encodes X at the optimum rate so that X can be reconstructed within a certain distortion (or, fidelity criterion) by the decoder. Wyner and Ziv computed the optimum rate R∗ Y (d) of transmitting X for the decoder to reconstruct X̂ that is within an average distortion d of X (for a specified distortion measure). The